분류기법을 활용한 정기 예금 고객 예측¶
Import Library¶
In [1]:
# 데이터 조작
import numpy as np
import pandas as pd
# 변수 검정
from scipy import stats
# 시각화
import matplotlib.pyplot as plt
import seaborn as sns
# 경고 숨김
import warnings
warnings.filterwarnings('ignore')
# 한글폰트 사용
plt.rc('font', family = 'Malgun Gothic')
plt.rc('axes', unicode_minus = False)
# 레티나 설정
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('retina')
Import Data¶
In [2]:
df = pd.read_csv('data/bank_customer_survey.csv')
print(df.shape)
df.head()
Out[2]:
Meaning of Columns¶
UCI Machine Learning Repository - Bank Marketing Data Set
<고객 개인 정보>
- age : 나이 (연속형, int)
- job : 직업 (범주형, obj)
- 노동자('blue'), 관리직('management'), 기술직('technician'), 관리직('admin'),
- 서비스직('services'), 은퇴('retired'), 자영업자('self'), 사업가('entrepreneur'),
- 무직('unemployed'), 주부('housemaid'), 학생('student'), 알 수 없음('unknown')
- marital : 결혼 여부 (범주형, obj)
- 이혼/사별('divorced'), 기혼('married'), 미혼('single')
- education : 교육 수준 (범주형, obj)
- 초등교육('primary'), 중등교육('secondary'), 고등교육(tertiary'), 알 수 있음('unknown')
<은행 관련 정보>
- balance : 잔고 (연속형, int)
- default : 채무 불이행 유무 (범주형, obj)
- 'no', yes'
- loan : 개인 대출 유무 (범주형, obj)
- 'no', 'yes'
- housing : 주택 담보 대출 유무 (범주형, obj)
- 'no', 'yes'
<텔레마케팅 관련 정보>
- contact : 연락 방법 (범주형, obj)
- 핸드폰('cellular'), 집전화('telephone'), 알 수 없음('unknown')
- duration : 총 통화 시간(초) (연속형, int)
- campaign : 이번 캠페인에 고객에게 연락한 수 (연속형, int)
- 마지막 연락까지 횟수에 포함
- day : 마지막으로 연락된 일 (연속형, int)
- month : 마지막으로 연락된 월 (범주형, obj)
- 1월('jan') ~ 12월('dec'),
- 2018년 5월 ~ 2010년 11월에 수집
- pdays : 고객이 이전 캠페인에 대해 연락을 받은 후 경과일 (연속형, int)
- '-1' : 이전 캠페인에 대해 연락을 받지 않음
<이전과 이번 마케팅 결과>
- previous : 캠페인 이전 연락 수 (연속형, int)
- poutcome : 이전 캠페인의 결과 (범주형, obj)
- 'failure', 'other', 'success', 'unknown'
- y : 정기 예금 설계 여부
- 0 : 'no', 1 : 'yes'
In [3]:
df.info()
EDA¶
결측치 확인¶
In [4]:
df.isnull().sum()
Out[4]:
In [5]:
# 'unknown' 도 결측치
print('job:', (df['job'] == 'unknown').sum())
print('education:', (df['education'] == 'unknown').sum())
print('contact:', (df['contact'] == 'unknown').sum())
print('poutcome:', (df['poutcome'] == 'unknown').sum())
85% 가 결측치이므로 poutcome 컬럼을 제거해야 합니다.
In [6]:
total = df['poutcome'].shape[0]
null = df.loc[(df['poutcome'] == 'unknown')|(df['poutcome'] == 'other'), 'poutcome'].shape[0]
print('결측치(%):', null / total * 100)
df['poutcome'].value_counts().plot.barh()
Out[6]:
중복값 확인¶
In [6]:
print(df.shape)
df = df[~df.duplicated()]
df.shape
Out[6]:
In [7]:
df.nunique()
Out[7]:
시각화¶
타겟 변수¶
캠페인 결과로 정기 예금을 들지 않은 고객('No')이 88%로 데이터 불균형 문제가 있습니다.
In [8]:
print(df['y'].value_counts(normalize=True))
sns.countplot(data=df, x='y')
Out[8]:
연속형 변수¶
age 와 day 외 다른 연속형 변수는 모두 이상치가 확인됩니다.
In [9]:
num_cols = df.dtypes[df.dtypes != 'object'].index.tolist()[:-1]
print(num_cols)
df.describe()
Out[9]:
age 외 대부분의 컬럼이 정규분포를 따르지 않아 차후 스케일링이 필요합니다.
In [10]:
h = df[num_cols].hist(figsize=(20, 15))
duration만 목표변수에 따라 큰 차이를 보이는 것 같아 보입니다.
In [11]:
num_cols.append('y')
sns.pairplot(df[num_cols], hue='y', diag_kws={'bw':5})
Out[11]:
In [12]:
numeric_features = [
'age',
'balance',
'duration',
'campaign',
'pdays',
'previous'
]
len(numeric_features)
Out[12]:
In [13]:
fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(15,10))
df[df.y == 0][numeric_features].hist(bins=35, color="blue", alpha=0.5, ax=axes)
df[df.y == 1][numeric_features].hist(bins=35, color="orange", alpha=0.7, ax=axes)
plt.legend(['No deposit', 'deposit'], shadow=True, loc=9)
Out[13]:
변수간 상관계수가 높은 변수는 pdays 와 previous 입니다.
In [14]:
corr = df[num_cols].corr()
sns.heatmap(data=corr, annot=True, cmap='seismic', vmax=1, vmin=-1)
Out[14]:
당연한 결과로 이어질 수 있는 'duration'변수는 삭제해야 합니다.
범주형 변수¶
In [15]:
cat_cols = df.dtypes[df.dtypes == 'object'].index.tolist()
cat_cols
Out[15]:
정기 예금을 신청한 고객 중에서는
- 결혼유무에선 기혼자가 비교적 많습니다.
- 교육수준에선 대졸자 이상이 비교적 많습니다.
- 채무 불이행 경험이 없는 사람이 있는 사람보다 많습니다.
- 주택 담보 대출을 받지 않은 사람이 받은 사람보다 많습니다.
- 개인 대출을 안 받은 사람이 받은 사람보다 많습니다.
- 개인 핸드폰으로 연락을 받은 사람이 더 많습니다.
반면에 정기 예금을 신청하지 않은 고객 중에서는
- 주택 담보 대출을 받은 사람이 받지 않은 사람보다 더 많습니다.
In [16]:
fig, axes = plt.subplots(ncols=3, nrows=3, figsize=(20, 20))
for i, col_name in enumerate(cat_cols):
row = i // 3
col = i % 3
sns.countplot(data=df, x=col_name, hue='y', ax=axes[row, col])
직업만 따로 보았습니다.
In [17]:
print(df[df.y==1]['job'].value_counts().sort_values(ascending=False))
plt.figure(figsize=(10, 5))
plt.xticks(rotation=45)
sns.countplot(data=df, x='job', hue='y')
Out[17]:
각각의 컬럼별 계좌 잔고에 따른 정기 예금 신청 결과의 분포를 보았습니다.
In [19]:
fig = plt.figure(figsize=(20, 15))
ax1 = fig.add_subplot(221)
ax2 = fig.add_subplot(222)
ax3 = fig.add_subplot(212)
g = sns.violinplot(data=df, x='default', y='balance', hue='y', ax=ax1)
g.set_title('채무 불이행 여부별 정기 예금 신청 결과')
g1 = sns.violinplot(data=df, x='education', y='balance', hue='y', ax=ax2)
g1.set_title('교육 수준별 정기 예금 신청 결과')
g2 = sns.violinplot(data=df, x='job', y='balance', hue='y', ax=ax3)
# g2.set_xticklabels(df['job'].unique(), rotation = 90, rotation_mode = 'anchor')
g2.set_title('직업별 정기 예금 신청 결과')
plt.show()
In [20]:
fig = plt.figure(figsize=(20, 15))
ax1 = fig.add_subplot(221)
ax2 = fig.add_subplot(222)
ax3 = fig.add_subplot(212)
g = sns.violinplot(data=df, x='loan', y='balance', hue='y', ax=ax1)
g.set_title('개인 대출 여부별 정기 예금 신청 결과')
g1 = sns.violinplot(data=df, x='housing', y='balance', hue='y', ax=ax2)
g1.set_title('주택 담보 대출 여부별 정기 예금 신청 결과')
g2 = sns.violinplot(data=df, x='marital', y='balance', hue='y', ax=ax3)
# g2.set_xticklabels(df['job'].unique(), rotation = 90, rotation_mode = 'anchor')
g2.set_title('결혼 여부별 정기 예금 신청 결과')
plt.show()
정기 예금 고객만 확인해보았습니다.
In [21]:
h = df[df['y'] == 1].hist(figsize=(15, 15))
In [22]:
cat = df[cat_cols]
cat_y = cat[df['y'] == 1]
fig, axes = plt.subplots(ncols=3, nrows=3, figsize=(15, 15))
for i, col_name in enumerate(cat_cols):
row = i // 3
col = i % 3
sns.countplot(data=cat_y, x=col_name, ax=axes[row, col])
직업별 교육수준을 확인해보았습니다.
In [23]:
plt.figure(figsize=(16, 8))
sns.countplot(data=df, x='education', hue='job')
Out[23]:
계좌 잔고가 높은 사람을 확인해보았습니다.
In [24]:
plt.figure(figsize=(15, 8))
sns.scatterplot(data=df, x='balance', y='age', hue='job')
Out[24]:
In [25]:
plt.figure(figsize=(15, 8))
sns.scatterplot(data=df, x='balance', y='age', hue='marital')
Out[25]:
In [26]:
plt.figure(figsize=(15, 8))
sns.scatterplot(data=df, x='balance', y='age', hue='education')
Out[26]:
데이터가 2008년 5월부터 2010년 11월까지 수집되었고 시작된 달에 가장 많이 캠페인 활동이 있었습니다.
In [27]:
df.month.value_counts().plot.bar()
Out[27]:
In [28]:
df_y1 = df[df['y']==1]
df_y1.month.value_counts().plot.bar()
Out[28]:
In [29]:
values = [[142, 1403], [441, 2649], [248, 477], [577, 2932], [925, 13766], [546, 5341], [627, 6895], [688, 6247], [269, 579], [323, 738], [403, 3970], [100, 214]]
index = ['jan', 'feb', 'mar', 'apr', 'may', 'jun', 'jul', 'aug', 'sep', 'oct', 'nov', 'dec']
columns = ['true', 'counts']
months = pd.DataFrame(values, index=index, columns=columns)
print(months)
각각의 달에 따른 정기 예금 신청 결과 비율을 보았습니다.
In [30]:
months['prob'] = months['true'] / months['counts']
months
Out[30]:
In [85]:
fig , ax0 = plt.subplots(figsize=(10, 10))
ax1 = ax0.twinx()
ax0.set_title("Month efficiency;")
ax0.plot(months["prob"] ,'r-', label = "prob" ,marker='o')
ax0.set_ylabel("prob")
ax0.grid(False)
ax1.plot(months["counts"] ,'b:', label ="count" , marker = 's')
ax1.set_ylabel("count")
ax1.grid(False)
ax0.set_xlabel("months")
fig.legend()
plt.show()
Feature Engineering¶
EDA를 통해 확인해보니 변수의 이상치 처리가 필요해보였습니다.
- 'pdays'변수는 -1인 값이 고객이 이전 캠페인에 대해 연락을 받지 않았음을 의미하는데 대부분의 값이 -1이라 이전 캠페인에 대한 응답여부 변수로 바꿔줄 것 입니다.
- 'previous'변수는 이번 캠페인 이전에 수행된 연락횟수를 의미하는데 과도하게 큰 이상치 하나를 지워줄 것입니다.
- 'day'변수는 이번 캠페인에서 마지막으로 연락된 날짜인데 분석에 의미가 없기 때문에 삭제할 것입니다.
- 종속변수와 높은 상관관계를 가져서 당연한 결과로 이어지는 'duration' 변수는 삭제할 것입니다.
- 'poutcome'변수는 이상치가 너무 많아서 삭제할 것입니다.
'pdays' 처리¶
In [32]:
df['panswer'] = np.where(df['pdays'] == -1, 0, 1)
df['panswer'].value_counts()
Out[32]:
In [33]:
del df['pdays']
'previous' 처리¶
In [34]:
df.previous.value_counts(ascending=True).head(3)
Out[34]:
In [35]:
df[df.previous == 275]
Out[35]:
In [36]:
pre_275 = df[df['previous'] == 275].index
df = df.drop(pre_275)
'day' 처리¶
In [37]:
del df['day']
'duration' 처리¶
In [38]:
del df['duration']
'poutcome' 처리¶
In [39]:
del df['poutcome']
변수 인코딩¶
In [40]:
df.to_csv('telemarketing.csv', index=False)
df = pd.read_csv('telemarketing.csv')
df.head()
Out[40]:
In [41]:
cat_columns = ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'month']
In [42]:
bin_cols = df.nunique()[df.nunique() == 2].index.tolist()
print(bin_cols)
In [43]:
bin_cols = ['default', 'housing', 'loan']
bin_cols
Out[43]:
In [44]:
multi_cols = [x for x in cat_columns if x not in bin_cols]
multi_cols
Out[44]:
In [45]:
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
for i in bin_cols:
df[i] = le.fit_transform(df[i])
df[bin_cols].head()
Out[45]:
In [46]:
df = pd.get_dummies(data = df, columns = multi_cols, drop_first=True)
df
Out[46]:
In [47]:
df.info()
변수 스케일링¶
'balance'변수는 이상치가 매우 크기 때문에 로그변환 후 standard scaling을 진행하였습니다.
In [48]:
df['balance'] = df['balance'] + 8020
In [49]:
df.balance.describe()
Out[49]:
In [50]:
import math
from sklearn import preprocessing
df['balance'] = preprocessing.scale(np.log(df['balance']+1))
df.describe()
Out[50]:
In [51]:
num_cols = ['balance', 'campaign', 'previous']
In [52]:
from sklearn.preprocessing import StandardScaler
std = StandardScaler()
scaled = std.fit_transform(df[num_cols])
scaled = pd.DataFrame(scaled, columns = num_cols)
scaled.head()
Out[52]:
In [53]:
print(df.shape)
df1 = df.drop(columns = num_cols, axis = 1)
df1 = df1.merge(scaled, left_index=True, right_index=True, how = "left")
print(df1.shape)
변수 설정 후 상관관계 확인¶
In [54]:
sns.set(rc={'figure.figsize':(12,10)})
sns.heatmap(df1.corr(), cmap="seismic", annot=False, vmin=-1, vmax=1)
Out[54]:
In [55]:
df1.to_csv('survey_features.csv', index=False)
df1 = pd.read_csv('survey_features.csv')
df1.head()
Out[55]:
의사결정나무¶
In [56]:
X, y = df1.drop('y',axis=1), df1['y']
import statsmodels.api as sm
# 절편 정의
X = sm.add_constant(X)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,
y,
test_size=0.2,
random_state=42)
In [57]:
from sklearn.tree import DecisionTreeClassifier, export_graphviz
# Dot to png
import pydot
# 의사결정 트리 선언
dTreeAll = DecisionTreeClassifier(random_state=0)
# 훈련 (모든 리프 노드 사용)
dTreeAll.fit(X_train, y_train)
# 점수 출력
print("Train Set Score1 : {:.2f}".format(dTreeAll.score(X_train, y_train)))
print("Test Set Score1 : {:.2f}".format(dTreeAll.score(X_test, y_test)))
# 의사결정 트리 선언(트리 깊이 제한)
dTreeLimit = DecisionTreeClassifier(max_depth=5, random_state=42)
# 훈련 (가지치기 : 리프노드 깊이 제한)
dTreeLimit.fit(X_train, y_train)
# 점수 출력
print("Train Set Score2 : {:.2f}".format(dTreeLimit.score(X_train, y_train)))
print("Test Set Score2 : {:.2f}".format(dTreeLimit.score(X_test, y_test)))
export_graphviz(dTreeLimit, out_file="dicisionTree1.dot", class_names=["No deposit","deposit"],
feature_names=df1.columns, impurity=False, filled=True)
# Encoding 중요
(graph,) = pydot.graph_from_dot_file('dicisionTree1.dot', encoding='utf8')
# Dot 파일을 Png 이미지로 저장
graph.write_png('dicisionTree1.png')
In [58]:
predictions = dTreeLimit.predict(X_test)
mf = dTreeLimit.feature_importances_
plt.figure(figsize=(10,5))
sns.barplot(y=X.columns, x= mf)
Out[58]:
In [101]:
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score
print('accuracy = {:.3f}'.format(accuracy_score(y_test, predictions) * 100))
print('f1_score = {:.3f}'.format(f1_score(y_test, predictions)))
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, predictions)
print(cm)
print(sns.heatmap(cm, annot=True, cmap='Blues'))
from sklearn.metrics import classification_report
cr = classification_report(y_test, predictions)
print(cr)
로지스틱 회귀분석¶
In [60]:
# fitting 로지스틱 회귀
import statsmodels.api as sm
pd.options.mode.chained_assignment = None
logitreg = sm.Logit(y_train, X_train)
logitreg_fit = logitreg.fit()
print(logitreg_fit.summary())
In [61]:
test_prob_logitreg = logitreg_fit.predict(X_test)
test_prob_logitreg
Out[61]:
후진제거법¶
In [62]:
cols = X.columns
In [63]:
def back_feature_elem (data_frame,dep_var,col_list):
""" Takes in the dataframe, the dependent variable and a list of column names, runs the regression repeatedly eleminating feature with the highest
P-value above alpha one at a time and returns the regression summary with all p-values below alpha"""
while len(col_list)>0 :
model=sm.Logit(dep_var,data_frame[col_list])
result=model.fit(disp=0)
largest_pvalue=round(result.pvalues,3).nlargest(1)
if largest_pvalue[0]<(0.05):
return result
break
else:
col_list=col_list.drop(largest_pvalue.index)
result=back_feature_elem(X,y,cols)
In [64]:
result.summary()
Out[64]:
오즈비 확인¶
In [65]:
params = np.exp(result.params)
conf = np.exp(result.conf_int())
conf['OR'] = params
pvalue=round(result.pvalues,3)
conf['pvalue']=pvalue
conf.columns = ['CI 95%(2.5%)', 'CI 95%(97.5%)', 'Odds Ratio','pvalue']
print ((conf))
'age'변수 라벨링하여 돌려보기¶
In [66]:
def age_split(df1) :
if (df1['age'] > 17) & (df1['age'] <= 33) :
return '18-33'
elif (df1['age'] > 33) & (df1['age'] <= 48) :
return '34-48'
elif (df1['age'] > 48) & (df1['age'] <= 95) :
return '49-95'
df1['age_group'] = df1.apply(age_split, axis = 1)
del df1['age']
In [68]:
df1
Out[68]:
In [74]:
age_group = ['age_group']
age_group
df1 = pd.get_dummies(data=df1, columns=age_group, drop_first=True)
df1
Out[74]:
In [75]:
X, y = df1.drop('y',axis=1), df1['y']
import statsmodels.api as sm
X = sm.add_constant(X)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,
y,
test_size=0.2,
random_state=42)
In [76]:
import statsmodels.api as sm
pd.options.mode.chained_assignment = None
logitreg = sm.Logit(y_train, X_train)
logitreg_fit = logitreg.fit()
print(logitreg_fit.summary())
In [77]:
cols = X.columns
def back_feature_elem (data_frame,dep_var,col_list):
""" Takes in the dataframe, the dependent variable and a list of column names, runs the regression repeatedly eleminating feature with the highest
P-value above alpha one at a time and returns the regression summary with all p-values below alpha"""
while len(col_list)>0 :
model=sm.Logit(dep_var,data_frame[col_list])
result=model.fit(disp=0)
largest_pvalue=round(result.pvalues,3).nlargest(1)
if largest_pvalue[0]<(0.05):
return result
break
else:
col_list=col_list.drop(largest_pvalue.index)
result=back_feature_elem(X,y,cols)
In [78]:
result.summary()
Out[78]:
In [79]:
params = np.exp(result.params)
conf = np.exp(result.conf_int())
conf['OR'] = params
pvalue=round(result.pvalues,3)
conf['pvalue']=pvalue
conf.columns = ['CI 95%(2.5%)', 'CI 95%(97.5%)', 'Odds Ratio','pvalue']
print ((conf))
인사이트 도출¶
마케팅 대상
- 주택담보대출과 개인대출이 없는 고객
- 이전 캠페인에서 응답한 고객
- 무직인 고객과 학생
- 싱글인 고객
- 핸드폰으로 응답한 고객
- 가장 효율적인 달은 3월
- 9월, 10월, 12월도 효율적입니다.
- 18세 이상 33세 이하의 고객을 타겟으로 하면 좋습니다.
랜덤포레스트¶
In [116]:
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=100 , n_jobs=-1 , random_state=42)
rf.fit(X_train , y_train)
Out[116]:
In [117]:
from sklearn.model_selection import RandomizedSearchCV
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 1000, num = 10)]
max_features = ['auto', 'sqrt']
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
min_samples_split = [2, 5, 10]
min_samples_leaf = [1, 2, 4]
bootstrap = [True, False]
random_grid = {'n_estimators': n_estimators,
'max_features': max_features,
'max_depth': max_depth,
'min_samples_split': min_samples_split,
'min_samples_leaf': min_samples_leaf,
'bootstrap': bootstrap}
random_grid
Out[117]:
In [119]:
rf_random = RandomizedSearchCV(estimator = rf,
param_distributions = random_grid,
n_iter = 3,
cv = 3,
random_state=42,
n_jobs = -1)
In [120]:
rf_random.fit(X_train, y_train)
Out[120]:
In [121]:
rf_random.best_params_
Out[121]:
In [122]:
rf_model = rf_random.best_estimator_
rf_model
Out[122]:
In [123]:
rf_model.fit(X_train , y_train)
Out[123]:
In [124]:
rf_output = rf_model.predict(X_test)
feature_importances = rf_model.feature_importances_
In [126]:
plt.figure(figsize = (15,10))
sns.barplot(x =feature_importances , y = X.columns)
Out[126]:
In [127]:
from sklearn.metrics import accuracy_score
accuracy_score(y_test, rf_output) * 100
Out[127]:
In [128]:
from sklearn.metrics import classification_report
cr = classification_report(y_test, rf_output)
print(cr)
예측에 좋은 성능을 보이는 머신러닝 모델 찾기¶
In [89]:
features = result.params.index.to_list()[1:]
df1 = df1[features]
In [90]:
df1
Out[90]:
In [92]:
y
Out[92]:
In [103]:
from sklearn.model_selection import KFold, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB, BernoulliNB, MultinomialNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import LinearSVC, SVC
import sklearn.metrics as metrics
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, log_loss, fbeta_score
from sklearn.metrics import auc, roc_curve, roc_auc_score, precision_recall_curve
In [93]:
X = df1
In [94]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print('X_train', X_train.shape)
print('y_train', y_train.shape)
print('X_test', X_test.shape)
print('y_test', y_test.shape)
모델 리포트¶
In [95]:
def model_report(model_name, model):
print('\nSearch for OPTIMAL THRESHOLD, vary from 0.0001 to 0.9999, fit/predict on train/test data')
# X_train, y_train 으로 모델을 학습하는 코드를 작성해 주세요
model.fit(X_train, y_train)
optimal_th = 0.5 # start with default threshold value
for i in range(0,3):
score_list = []
print('\nLooping decimal place', i+1)
th_list = [np.linspace(optimal_th-0.4999, optimal_th+0.4999, 11),
# eg [ 0.0001 , 0.1008, 0.2006, 0.3004, 0.4002, 0.5, 0.5998, 0.6996, 0.7994, 0.8992, 0.9999 ]
np.linspace(optimal_th-0.1, optimal_th+0.1, 21),
# eg 0.3xx [ 0.2 , 0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.3 , 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.4 ]
np.linspace(optimal_th-0.01, optimal_th+0.01, 21)]
# eg 0.30x [ 0.29 , 0.291, 0.292, 0.293, 0.294, 0.295, 0.296, 0.297, 0.298, 0.299, 0.3 , 0.301, 0.302, 0.303, 0.304, 0.305, 0.306, 0.307, 0.308, 0.309, 0.31 ]
for th in th_list[i]:
y_pred = (model.predict_proba(X_test)[:,1] >= th) # predict_proba 로 예측하게 합니다.
f1scor = f1_score(y_test, y_pred) # y_test, y_pred 로 f1_score를 구해주세요.
score_list.append(f1scor)
print('{:.3f}->{:.4f}'.format(th, f1scor), end=', ') # display score in 4 decimal pl
optimal_th = float(th_list[i][score_list.index(max(score_list))])
print('optimal F1 score = {:.4f}'.format(max(score_list)))
print('optimal threshold = {:.3f}'.format(optimal_th))
print(model_name, 'accuracy score is')
print('Training: {:.2f}%'.format(100*model.score(X_train, y_train))) # score uses accuracy
print('Test set: {:.2f}%'.format(100*model.score(X_test, y_test))) # should use cross validation
y_pred = (model.predict_proba(X_test)[:,1] >= 0.25)
print('\nAdjust threshold to 0.25:')
print('Precision: {:.4f}, Recall: {:.4f}, F1 Score: {:.4f}'.format(
precision_score(y_test, y_pred), recall_score(y_test, y_pred), f1_score(y_test, y_pred)))
print(model_name, 'confusion matrix: \n', confusion_matrix(y_test, y_pred))
y_pred = model.predict(X_test)
print('\nDefault threshold of 0.50:')
print('Precision: {:.4f}, Recall: {:.4f}, F1 Score: {:.4f}'.format(
precision_score(y_test, y_pred), recall_score(y_test, y_pred), f1_score(y_test, y_pred)))
print(model_name, 'confusion matrix: \n', confusion_matrix(y_test, y_pred))
y_pred = (model.predict_proba(X_test)[:,1] >= 0.75)
print('\nAdjust threshold to 0.75:')
print('Precision: {:.4f}, Recall: {:.4f}, F1 Score: {:.4f}'.format(
precision_score(y_test, y_pred), recall_score(y_test, y_pred), f1_score(y_test, y_pred)))
print(model_name, 'confusion matrix: \n', confusion_matrix(y_test, y_pred))
y_pred = (model.predict_proba(X_test)[:,1] >= optimal_th)
print('\nOptimal threshold {:.3f}'.format(optimal_th))
print('Precision: {:.4f}, Recall: {:.4f}, F1 Score: {:.4f}'.format(
precision_score(y_test, y_pred), recall_score(y_test, y_pred), f1_score(y_test, y_pred)))
print(model_name, 'confusion matrix: \n', confusion_matrix(y_test, y_pred))
global model_f1, model_auc, model_ll, model_roc_auc
model_f1 = f1_score(y_test, y_pred)
y_pred = model.predict_proba(X_test)
model_ll = log_loss(y_test, y_pred)
print(model_name, 'Log-loss: {:.4f}'.format(model_ll))
y_pred = model.predict(X_test)
model_roc_auc = roc_auc_score(y_test, y_pred)
print(model_name, 'roc_auc_score: {:.4f}'.format(model_roc_auc))
y_pred = model.predict_proba(X_test)[:,1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
model_auc = auc(fpr, tpr)
print(model_name, 'AUC: {:.4f}'.format(model_auc))
# plot the ROC curve
plt.figure(figsize = [6,6])
plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % model_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic')
plt.legend(loc="lower right")
# plt.savefig('roc_auc_score')
plt.show()
return
# initialise lists to collect the results to plot later
model_list = []
f1_list = []
auc_list = []
ll_list = []
roc_auc_list = []
time_list = []
GaussianNB¶
In [97]:
import time
print('\n"""""" GaussianNB """"""')
time1 = time.time()
gnb = GaussianNB()
model_report('GaussianNB', gnb) # 위에서 작성한 모델 리포트 함수를 부릅니다.
model_list.append('GaussianNB')
f1_list.append(model_f1)
auc_list.append(model_auc)
ll_list.append(model_ll)
roc_auc_list.append(model_roc_auc)
time_list.append(time.time() - time1)
BernoulliNB¶
In [98]:
print('\n"""""" BernoulliNB """"""')
time1 = time.time()
bnb = BernoulliNB()
model_report('BernoulliNB', bnb) # 위에서 작성한 모델 리포트 함수를 부릅니다.
model_list.append('BernoulliNB')
f1_list.append(model_f1)
auc_list.append(model_auc)
ll_list.append(model_ll)
roc_auc_list.append(model_roc_auc)
time_list.append(time.time() - time1)
LogisticRegression¶
In [99]:
print('\n"""""" LogisticRegression """"""')
print('\nSearch for optimal hyperparameter C in LogisticRegresssion, vary C from 0.001 to 1000, using KFold(5) Cross Validation on train data')
kf = KFold(n_splits=5, random_state=21, shuffle=True) #produce the k folds
score_list = []
c_list = 10**np.linspace(-3,3,200)
for c in c_list:
logit = LogisticRegression(C = c)
cvs = (cross_val_score(logit, X_train, y_train, cv=kf, scoring='f1')).mean()
score_list.append(cvs)
print('{:.4f}'.format(cvs), end=", ") # 4 decimal pl
print('optimal cv F1 score = {:.4f}'.format(max(score_list)))
optimal_c = float(c_list[score_list.index(max(score_list))])
print('optimal value of C = {:.3f}'.format(optimal_c))
time1 = time.time()
logit = LogisticRegression(C = optimal_c)
model_report('LogisticRegression', logit)
model_list.append('LogisticRegression')
f1_list.append(model_f1)
auc_list.append(model_auc)
ll_list.append(model_ll)
roc_auc_list.append(model_roc_auc)
time_list.append(time.time() - time1)
KNN¶
In [111]:
print('\n"""""" KNN """""" (quite slow)')
print('\nSearch for optimal hyperparameter K in KNN, vary K from 1 to 20, using KFold(5) Cross Validation on train data')
kf = KFold(n_splits=5, random_state=21, shuffle=True) #produce the k folds
k_scores = []
for k in range(1, 21):
knn = KNeighborsClassifier(n_neighbors = k)
cvs = cross_val_score(knn, X_train, y_train, cv=kf, scoring='f1').mean()
k_scores.append(cvs)
print('{:.4f}'.format(cvs), end=", ")
print('optimal cv F1 score = {:.4f}'.format(max(k_scores))) # 4 decimal pl
optimal_k = k_scores.index(max(k_scores))+1 # index 0 is for k=1
print('optimal value of K =', optimal_k)
time1 = time.time()
knn = KNeighborsClassifier(n_neighbors = optimal_k)
model_report('KNN', knn)
print('\nCompare with KNN classification_report (same as default threshold 0.50)')
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
print(metrics.classification_report(y_test, y_pred))
model_list.append('KNN')
f1_list.append(model_f1)
auc_list.append(model_auc)
ll_list.append(model_ll)
roc_auc_list.append(model_roc_auc)
time_list.append(time.time() - time1)
DecisionTreeClassifier¶
In [104]:
print('\n"""""" DecisionTreeClassifier """"""')
print('\nSearch for optimal max_depth in DecisionTree, vary 2 to 10, using KFold(5) Cross Validation on train data')
kf = KFold(n_splits=5, random_state=21, shuffle=True) # produce the k folds
d_scores = []
for d in range(2, 11):
decisiontree = DecisionTreeClassifier(max_depth=d)
cvs = cross_val_score(decisiontree, X_train, y_train,
cv=kf, scoring='f1').mean()
d_scores.append(cvs)
print('{:.4f}'.format(cvs), end=", ")
print('optimal F1 score = {:.4f}'.format(max(d_scores))) # 4 decimal pl
optimal_d = d_scores.index(max(d_scores))+2 # index 0 is for d=2
print('optimal max_depth =', optimal_d)
time1 = time.time()
decisiontree = DecisionTreeClassifier(max_depth=optimal_d)
model_report('DecisionTreeClassifier', decisiontree)
model_list.append('DecisionTreeClassifier')
f1_list.append(model_f1)
auc_list.append(model_auc)
ll_list.append(model_ll)
roc_auc_list.append(model_roc_auc)
time_list.append(time.time() - time1)
RandomForestClassifier¶
In [105]:
print('\n"""""" RandomForestClassifier """""" (quite slow)')
print('\nSearch for optimal n_estimators in RandomForest, vary 100 to 500, using KFold(5) Cross Validation on train data')
kf = KFold(n_splits=5, random_state=21, shuffle=True) #produce the k folds
score_list = []
n_list = []
for n in [100, 150, 200, 250, 300, 350, 400, 450, 500]:
randomforest = RandomForestClassifier(n_estimators=n)
cvs = (cross_val_score(randomforest, X_train, y_train, cv=kf, scoring='f1')).mean()
score_list.append(cvs)
n_list.append(n)
print('{:.0f}->{:.4f}'.format(n, cvs), end=", ") # display score in 4 decimal place
print('optimal F1 score = {:.4f}'.format(max(score_list)))
optimal_n = int(n_list[score_list.index(max(score_list))])
print('optimal n_estimators = {:.0f}'.format(optimal_n))
time1 = time.time()
randomforest = RandomForestClassifier(n_estimators=optimal_n)
model_report('RandomForestClassifier', randomforest)
model_list.append('RandomForestClassifier')
f1_list.append(model_f1)
auc_list.append(model_auc)
ll_list.append(model_ll)
roc_auc_list.append(model_roc_auc)
time_list.append(time.time() - time1)
LinearSVC¶
In [106]:
print('\n"""""" LinearSVC """"""')
time1 = time.time()
linearsvc = LinearSVC()
# model_report('LinearSVC', linearsvc) # model has no attribute 'predict_proba'
linearsvc.fit(X_train, y_train)
print('LinearSVC accuracy score is')
print('Training: {:.2f}%'.format(100*linearsvc.score(X_train, y_train))) # score uses accuracy
print('Test set: {:.2f}%'.format(100*linearsvc.score(X_test, y_test))) # should use cross validation
y_pred = linearsvc.predict(X_test)
print(metrics.classification_report(y_test, y_pred))
print('LinearSVC confusion matrix: \n', confusion_matrix(y_test, y_pred))
model_f1 = f1_score(y_test, y_pred)
model_ll = log_loss(y_test, y_pred)
print('LinearSVC Log-loss: {:.4f}'.format(model_ll))
model_roc_auc = roc_auc_score(y_test, y_pred)
print('LinearSVC roc_auc_score: {:.4f}'.format(model_roc_auc))
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
model_auc = auc(fpr, tpr)
print('LinearSVC AUC: {:.4f}'.format(model_auc))
# plot the ROC curve
plt.figure(figsize = [6,6])
plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % model_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic')
plt.legend(loc="lower right")
# plt.savefig('roc_auc_score')
plt.show()
model_list.append('LinearSVC')
f1_list.append(model_f1)
auc_list.append(model_auc)
ll_list.append(model_ll)
roc_auc_list.append(model_roc_auc)
time_list.append(time.time() - time1)
SVC¶
In [107]:
print('\n"""""" SVC """""" (extremely slow)')
time1 = time.time()
svc = SVC(gamma='scale', probability=True)
model_report('SVC', svc)
model_list.append('SVC')
f1_list.append(model_f1)
auc_list.append(model_auc)
ll_list.append(model_ll)
roc_auc_list.append(model_roc_auc)
# time_list.append(time.time() - time1) # use this line for actual time spent, or
time_list.append(0) # use this line to be able to see time spent for other models
classification report scores¶
In [113]:
## plot the classification report scores
fig, ax = plt.subplots(5, 1, figsize=(18, 28))
# fig.set_figwidth(10)
# fig.set_figheight(6)
# fig.suptitle('Main Title',fontsize = 16)
ax[0].bar(model_list, f1_list)
ax[0].set_title('F1-score')
ax[1].bar(model_list, auc_list)
ax[1].set_title('AUC-score');
ax[2].bar(model_list, ll_list)
ax[2].set_title('Log-Loss-Score')
ax[3].bar(model_list, roc_auc_list)
ax[3].set_title('ROC AUC Score')
ax[4].bar(model_list, time_list)
ax[4].set_title('Time taken')
# Fine-tune figure; make subplots farther from each other, or nearer to each other.
fig.subplots_adjust(hspace=0.2, wspace=0.2)
In [114]:
model_list
Out[114]:
ROC curves¶
In [115]:
# plot the ROC curves
plt.figure(figsize=(10,10))
y_pred = gnb.predict_proba(X_test)[:,1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
plt.plot(fpr, tpr, color='blue',
lw=3, label='GaussianNB (area = %0.2f)' % auc_list[0])
y_pred = bnb.predict_proba(X_test)[:,1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
plt.plot(fpr, tpr, color='green',
lw=3, label='BernoulliNB (area = %0.2f)' % auc_list[1])
y_pred = logit.predict_proba(X_test)[:,1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
plt.plot(fpr, tpr, color='red',
lw=2, label='LogisticRegression (area = %0.2f)' % auc_list[2])
y_pred = decisiontree.predict_proba(X_test)[:,1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
plt.plot(fpr, tpr, color='purple',
lw=2, label='DecisionTree (area = %0.2f)' % auc_list[3])
y_pred = randomforest.predict_proba(X_test)[:,1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
plt.plot(fpr, tpr, color='brown',
lw=2, label='RandomForest (area = %0.2f)' % auc_list[4])
y_pred = linearsvc.predict(X_test)
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
plt.plot(fpr, tpr, color='cyan',
lw=2, label='LinearSVC (area = %0.2f)' % auc_list[5])
y_pred = svc.predict_proba(X_test)[:,1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
plt.plot(fpr, tpr, color='magenta',
lw=2, label='SVC (area = %0.2f)' % auc_list[6])
y_pred = knn.predict_proba(X_test)[:,1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
plt.plot(fpr, tpr, color='yellow',
lw=3, label='KNN (area = %0.2f)' % auc_list[7])
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.xlabel('False Positive Rate', fontsize=13)
plt.ylabel('True Positive Rate', fontsize=14)
plt.title('Receiver Operating Characteristic', fontsize=17)
plt.legend(loc='lower right', fontsize=13)
plt.show()
